Distributed Gradient Methods with Variable Number of Working Nodes

Distributed second order methods with variable number of working nodes

February 21, 2017 Abstract Recently, an idling mechanism has been introduced in the context of distributed first order methods for minimization of a sum of nodes’ local convex costs over a generic, connected network. With the idling mechanism, each node i, at each iteration k, is active – updates its solution estimate and exchanges messages with its network neighborhood – with probability pk (p...

Variable Metric Conjugate Gradient Methods

Distributed Delayed Proximal Gradient Methods

We analyze distributed optimization algorithms where parts of data and variables are distributed over several machines and synchronization occurs asynchronously. We prove convergence for the general case of a nonconvex objective plus a convex and possibly nonsmooth penalty. We demonstrate two challenging applications, `1-regularized logistic regression and reconstruction ICA, and present experi...

Steepest Descent and Conjugate Gradient Methods with Variable Preconditioning

We analyze the conjugate gradient (CG) method with variable preconditioning for solving a linear system with a real symmetric positive definite (SPD) matrix of coefficients A. We assume that the preconditioner is SPD on each step, and that the condition number of the preconditioned system matrix is bounded above by a constant independent of the step number. We show that the CG method with varia...

Distributed Accelerated Proximal Coordinate Gradient Methods

We develop a general accelerated proximal coordinate descent algorithm in distributed settings (DisAPCG) for the optimization problem that minimizes the sum of two convex functions: the first part f is smooth with a gradient oracle, and the other one Ψ is separable with respect to blocks of coordinate and has a simple known structure (e.g., L1 norm). Our algorithm gets new accelerated convergen...